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FOREWORD 

The  lack  of  meaningful  measures  of  effectiveness  for  the  Quality 
Assurance  (QA)  function  within  the  Defense  Contract 
Administrative  Services  (DCAS)  has  been  a  serious  deficiency  for 
many  years.  With  the  availability  of  new,  automated  data  from 
the  QA  Management  information  System,  these  measures  are  made 
possible  through  the  QUality  Effectiveness  Sensing  Technique 
(QUEST)  model  which  is  documented  in  this  report. 

QUEST  evaluates  both  the  Government- driven  Contract  QA  Program 
and  the  contractor-driven  product  conformance  through  a  set  of 
indicators  using  multi  -  attribute  decision-making  methods.  These 
techniques  combine  quantitative  statistical  analysis  with 
subjective  factors  provided  by  QA  experts.  The  model  was 
successfully  tested  by  comparing  QUEST  measures  with  supervisor's 
opinions  throughout  DCAS  organizational  elements.  It  is, 
therefore,  recommended  that  QUEST  measures  be  implemented,  as 
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EXECUTIVE  SUMMARY 


Defense  Contract  Administrative  Services  (DCAS)  organizations  have  used 
measures  of  efficiency  for  several  years  to  determine  the  productivity  of  its 
Quality  Assurance  (QA)  functions.  However,  DCAS  has  lacked  uniform  measures 
of  effectiveness  (MOE)  concerning  the  degree  of  mission  accomplishment  for  QA. 
With  the  implementation  of  a  new  automated  data  processing  system,  the 
Mechanization  of  Contract  Administration  Services,  Phase  II,  data  now  exists 
which  can  produce  such  measures.  The  objectives  of  this  study  effort  were  to 
define  QA  effectiveness,  to  identify  detailed  MOE,  and  to  develop  a  model 
which  produces  both  detailed  and  aggregated  MOE. 

With  the  assistance  of  DCAS  personnel  and  Study  Advisory  Group  members, 
the  DLA  Operations  Research  and  Economic  Analysis  Office  has  developed  the 
QUality  Effectiveness  Sensing  Technique  (QUEST)  model.  QUEST  measures  QA 
effectiveness  based  on  a  dual  definition  of  effectiveness.  QA  is  effective 
when  the  Contract  Quality  Assurance  Program  (CQAP)  is  operated  in  accordance 
with  established  policies  and  procedures  and  the  product  is  produced  in 
accordance  with  contract  provisions. 

To  measure  program  and  product  effectiveness,  facilities  (contractors)  are 
peer  grouped  based  on  commodity,  QA  provision,  and  size.  Facility  data  are 
compared  against  peer  group  averages  and  standard  deviations  to  produce  a 
relative  measure.  QUEST  measures,  therefore,  represent  how  well  the 
government  QA  representative  (QAR)  and  the  contractor  together  compare  with 
others  in  similar  circumstances. 

Program  effectiveness  measures  are  primarily  a  function  of  the  actions  of 
the  QAR.  QUEST  evaluates  17  CQAP  functions  and  identifies  "red  flags"  when  an 
out-of-tolerance  condition  is  detected.  Flags  can  occur  when  the  QAR  fails  to 
perform  expected  actions  or  when  the  actions  performed  are  abnormal  in 
relation  to  peers.  QUEST  counts  flags  and  scores  program  effectiveness  based 
on  the  number  of  flags  generated  per  facility  per  month. 

Product  effectiveness  is  measured  using  a  multi-attribute  decision-making 
technique  called  Technique  for  Order  Preference  by  Similarity  to  Ideal 
Solution  (TOPSIS) .  This  technique  has  been  successfully  used  by  the  Army  for 
officer  promotion  boards,  by  DLA  to  determine  which  items  require  Industrial 
Preparedness  Planning  and  in  many  other  applications.  Its  unique  concept  of 
defining  best  (ideal)  and  worst  (negative  ideal)  scenarios  as  a  reference  for 
measurement  is  less  subjective  than  many  other  multi-attribute  techniques. 
Seven  product-related  indicators  and  their  trends  are  measured  individually 
and  as  a  group  to  achieve  a  product  MOE.  These  product  indicators  are 
primarily  driven  by  the  contractor's  performance  but  can  be  influenced  by  the 
QAR . 


QUEST  provides  an  accurate  measure  of  QA  effectiveness,  as  demonstrated  by 
the  successful  validation  achieved  by  comparing  model  results  with  expert 
opinion  obtained  by  questionnaires.  Therefore,  it  is  recommended  that  the 
QUEST  model  be  implemented  by  the  DLA  Directorate  of  Quality  Assurance  in 
accordance  with  a  plan  developed  by  the  QUEST  Study  Advisory  Group. 


INTRODUCTION 


I . 


A .  Background 

The  Defense  Contract  Administrative  Services  (DC AS)  administers  a  wide  range 
of  contracts  for  the  Military  Services,  Defense  agencies  and  other  Federal 
agencies.  In-plant  quality  assurance  (QA)  responsibilities  are  implemented  hy 
DCAS  through  the  Contract  Quality  Assurance  (CQA)  program  (CQAP) .  This  five 
part  program  is  specified  in  DLAM  8200.1,  Defense  In-Plant  Quality  Assurance 
Program,  and  DLAM  8200.2,  Procurement  Quality  Assurance  Support  Manual  for 
Defense  Contract  Administrative  Services. 

The  QA  Management  Information  System  (QAMIS)  enables  DCAS  to  manage  and 
control  certain  aspects  of  the  CQA  program.  Through  data  reported  by 
individual  QA  representatives  (QARs) ,  information  relating  to  contracts, 
contractors  and  QAR  actions  is  systematically  collected,  reported  and 
summarized  by  QAMIS.  Prior  to  1986  all  regions  utilized  a  batch  process  mode 
of  QAMIS  whereby  the  QAR-generated  forms  were  mailed  to  automated  data 
processing  (ADP)  operations  and  monthly  hard  copy  reports  were  generated. 
Beginning  at  DCAS  region  (DCASR)  Atlanta  (DCASR  ATL) ,  an  improved  version  of 
QAMIS  was  established  as  part  of  the  Mechanization  of  Contract  Administrative 
Service  (MOCAS) ,  Phase  II  Segment  IX.  In  addition  to  on-line  data  input,  on¬ 
line  query  capability,  and  other  operational  improvements,  more  detailed  data 
elements  are  reported,  providing  better  visibility  of  CQA  operations.  This 
new  system  is  now  operational  at  DCASR-Chicago  (DCASR  CHI) ,  DCASR-Cleveland 
(DCASR  CLE),  DCASR-Dallas  (DCASR  DAL)  and  DCASR-St.  Louis  (DCASR  STL).  Plans 
are  underway  to  extend  the  new  QAMIS  to  the  other  regions. 

As  a  result  of  the  new  data  elements  available  under  the  new  QAMIS,  the  DLA 
Operations  Research  and  Economic  Analysis  Office  (DLA-LO)  was  tasked  to 
develop  a  model  that  produces  measures  of  eflectiven ess  of  the  DCAS  CQA 
program . 

B.  Problem  Statement.  To  develop  a  methodology  to  measure  the  relative 
effectiveness  of  CQA  operations  from  the  contractor  facility  level  to  the 
regional  level. 

C .  Objectives 

1.  To  define  CQA  "effectiveness." 

2.  To  define  measures  of  CQA  effectiveness  that  are  compatible  with 
available  data. 

3.  To  develop  analytical  methodologies  to  compute  measures  of 
effectiveness . 

A.  To  develop  a  methodology  to  combine  individual  measures  of 

effectiveness  into  a  consistent  measure  of  overall  relative  effectiveness. 
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D.  Scope 


1.  Measures  of  effectiveness  are  limited  to  using  those  data 
elements  reported  under  QAMIS,  Phase  II.  No  new  or  additional  reporting 
burdens  will  be  generated  to  support  this  effort. 

2.  Model  development  is  limited  to  data  bases  established  at  the 
five  Phase  II  DCAS  regions  beginning  in  January  1986  through  July  1987. 

E.  Approach .  Extensive  use  was  made  of  experienced  supervisory  and 
staff  experts  at  the  DCASRs  and  DLA  Directorate  of  Quality  Assurance. 
Brainstorming  sessions  with  DCASR  personnel  produced  the  input  resulting  in 
accomplishing  the  first  two  objectives  specified  in  paragraph  IC.  A  Study 
Advisory  Group  (SAG),  shown  in  Appendix  A,  was  formed  to  screen  and  augment 
the  input  from  the  brainstorming  sessions,  to  provide  subjective  input  such  as 
weighting  factors  and  to  guide  and  assist  the  analysts  in  conceptual 
development.  The  SAG  also  assisted  in  the  validation  of  the  model  and  in 
developing  a  plan  to  implement  the  model.  The  modelling  approach  was  designed 
to  emulate  the  thought  processes  of  knowledgeable  experts  as  they  subjectively 
and  objectively  view  the  effectiveness  of  QA. 

II.  METHODOLOGY .  QUEST  is  a  FORTRAN  based  model.  Its  operation  is  designed 
for  an  IBM  operating  environment,  using  certain  IBM  utility  programs  and  IBM 
JCL.  Generally,  QUEST  extracts  relevant  data  from  QAMIS  and  processes  the 
information  systematically  to  produce  both  detailed  and  summary  level  measures 
of  effectiveness.  This  is  done  by  first  assigning  each  facility  to  a  peer 
group.  Then  monthly  QAMIS  data  for  that  facility  are  compared  to  its 
historical  peer  group  averages  to  assess  the  relative  effectiveness  of  the  CQA 
program  and  the  potential  for  product  non-conformance. 

A.  Data  Sources .  Raw  data  to  support  this  effort  are  needed  from  four 
sources .  Data  records  for  the  QUEST  model  are  developed  by  merging  data 
elements  from  these  sources  by  matching  the  Commercial  and  Government  Entity 
(CAGE)  codes  (formerly  Federal  Supply  Codes  for  Manufacturers  (FSCM)).  The 
term  FSCM  will  be  used  throughout  the  remainder  of  this  report  to  refer  to 
facilities . 

1.  Performance  History  File.  Each  month  a  Phase  II  DCASR  produces 
a  tape  containing  records  for  each  FSCM  reported  under  QAMIS.  The  reported 
elements  are  provided  in  Appendix  B  and  defined  in  DLAM  8200.2.  Generally, 
these  data  provide  hours,  counts  and  statistics  pertaining  to  QA  activities 
during  that  month. 

2.  Facility  Profile.  For  each  FSCM  under  CQAP,  the  QAR  enters 
descriptive  data  about  that  facility  when  that  facility  is  first  monitored. 
As  conditions  change,  the  QAR  updates  the  Facility  Profile.  Each  month  a 
Phase  II  DCASR  produces  a  Facility  Profile  Tape  as  shown  in  Appendix  B. 
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3.  Materiel  Deficiency  Report  (MDR)  File.  Each  Phase  II  DCASR 
maintains  an  on-line  file  for  monitoring  Materiel  Deficiency  Reports.  These 
reports  and  available  data  elements  are  described  in  DLAR  8260.2,  Materiel 
Deficiency  Investigation  and  Reporting  System.  Because  cf  the  file  structure 
of  the  on-line  MDR  file,  a  special  program  is  required  to  produce  a  "flat" 
file  of  fixed  length  records  needed  by  QUEST.  This  utility  based  program  must 
be  run  each  month  prior  to  running  the  QUEST  model  to  produce  a  file  shown  in 
Appendix  B. 

4.  Contractor  Alert  File.  Each  month  DCASR  Boston  issues  a  list  of 
contractors  that  have  a  history  of  past  problems.  The  purpose  of  the  alert 
file  is  to  warn  acquisition  elements  that  a  DCASR  has  experienced  problems  and 
to  provide  data  concerning  the  nature  of  these  problems .  Approximately  1000 
FSCMs  currently  appear  on  the  alert  list  nationwide.  An  "alert"  can  be  issued 
because  of  quality,  production,  technical  or  financial  problems. 

B.  Data  Base  Establishment.  Prior  to  execution  of  the  QUEST  model,  a 
data  base  must  be  established  and  maintained  as  follows. 

1.  MDR  Processor .  The  purpose  of  this  module  is  to  identify 
"valid"  MDRs ,  assign  weighting  factors  to  reflect  the  "age"  of  the  MDR  and  to 
prepare  the  MDR  record  for  merging  with  the  history  file.  This  record 
contains,  for  each  facility,  the  weighted  number  of  valid  MDRs  and  the  average 
days  to  close  an  MDR  per  month.  See  Appendix  C  for  additional  details. 

2.  File  Merger.  A  module  documented  in  Appendix  D  is  used  to 
create  a  historical  data  base  of  up  to  120  months  of  data  on  tape.  The  input 
files  must  be  sorted  by  FSCM  by  year  by  month  in  ascending  order  prior  to  this 
step.  Because  of  the  dynamic  nature  of  the  MDR  file  (also  to  a  lesser  degree 
the  performance  history  file  and  the  facility  profile),  it  will  be  necessary 
to  not  only  merge  the  current  month  into  the  d>. ’.a  base,  but  also  to  recreate 
several  past  months  each  month.  Appended  to  the  performance  history  file  are 
the  following: 


a.  Number  of  government  QA  personnel  assigned  to  the  faci  lity 
from  the  Facility  Profile. 

b.  Contractor  Operation  Type  Codes  (i.e.,  manufacturer, 

overhaul/repair,  dis tr ibutor/ j obber ,  etc.)  from  the  Facility  Profile. 

c.  Weighted  MDR  count  from  the  MDR  processor. 

d.  Average  days  to  close  MDRs  from  the  MDR  processor. 

C.  Data  Selection.  This  section  describes  how  the  QUEST  model  selects 
data  for  processing  from  the  tape  data  file  described  in  the  previous  section. 
A  screening  process  is  detailed  in  Appendix  E. 

1.  Program  Control.  A  small  input  file,  documented  in  Appendix  E, 
must  be  created  to  control  the  execution  of  the  QUEST  model.  This  file 
specifies  parameters  used  in  the  screening  process. 
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2.  Record  Selection.  Data  records  from  the  History  Tape  are 
screened  to  determine  if  further  processing  is  in  order.  These  selected 
records  are  stripped  of  extraneous  data  elements  and  put  into  a  disk  file 
Reasons  for  non- selection  are  inactivity,  invalid  or  missing  data  and  dates 
outside  the  horizons  specified  for  model  execution.  Appendix  E  further 
describes  this  process. 

D.  Stratification  Process.  To  compute  effectiveness  measures  QUEST 
interprets  QAMIS  data  by  comparing  raw  data  against  average  values  for  similar 
facilities.  The  deviation  from  average  translates  into  a  relative 
effectiveness  score.  QUEST  measures  of  effectiveness  are,  in  reality, 
measures  of  facilities  relative  to  a  peer  grouping  established  by  QUEST  model 
logic  based  on  facility  size,  dominant  commodity  and  QA  provision.  QUEST 
assigns  a  peer  group  number  to  each  facility  evaluated.  This  grouping  process 
is  described  in  general  below.  Appendix  F  provides  details  on  grouping 
methodology  and  additional  documentation  is  shown  in  Appendix  E,  pages  E-9 
through  E-12. 

1.  Grouping  Process.  Criteria  for  grouping  facilities  was 
constrained  by  the  need  to  have  a  sufficient  number  of  facilities  to  compute 
reliable  averages  and  standard  deviations  for  QAMIS  data  elements.  Thus 
grouping  was  accomplished  in  two  steps.  Using  a  theoretical,  unconstrained 
logic  the  facilities  were  grouped.  Based  on  the  unconstrained  grouping,  a 
statistical  analysis  was  run  to  count  the  number  of  observations  in  each 
group.  When  the  number  of  observations  was  small,  logical  groupings  of  "low 
hit"  groups  were  sought  to  combine  into  a  larger  group. 


2.  Grouping  Logic.  A  theoretical  logic  to  group  facilities  was 
designed  and  approved  by  the  SAG.  However,  it  became  necessary  to  depart  from 
theory  because  of  the  extreme  lack  of  homogeneity  of  groups.  Some  groups  have 
hundreds  of  "peers;"  others  have  very  few  or  none.  A  two  step  process  used  to 
overcome  this  problem  is  described  as  follows: 


a.  Unconstrained  Groups 


(1)  Resident  Facilities.  Because  of  the  limited  number  of 
Resident  facilities,  only  240  groupings  (16x3x5)  were  attempted  based  on: 

(a)  Commodity  Code.  The  first  alpha  commodity  code  is 
selected.  Commodity  code  is  a  two  alpha  code  where  the  first  alpha  is  the 
primary  commodity  designator  and  the  second  alpha  further  defines  the 
commodity  within  the  primary  grouping.  Commodity  Codes  are  defined  in  DLAM 
8200.2.  There  are  16  primary  commodity  codes. 

(b)  Quality  Provision.  Grouping  by  QA  Provision  is 
necessary  because  of  significant  differences  in  QA  activity  caused  by  the 
provision.  Three  provision  codes  are  used  for  MIL-Q-9858A  (MILQ) ,  MIL-I- 
45208A  (MILI)  and  standard  inspection  provisions. 


(c)  Facility  Size.  Size  is  measured  by  the  number  of 
Government  QA  personnel  assigned  to  the  facility.  Five  size  groupings  were 
selected  as  follows: 
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I  0  -  2  QA  Specialists  (QAS) 


2  3  -  7  QAS 

3  8-20  QAS 

4  Reserved  for  future  use 

5  Over  20  QAS 

(2)  Nonresident  Facilities.  Because  there  are  more 
nonresident  facilities  than  resident  facilities,  more  detailed  groupings  were 
feasible.  There  were  384  groups  (16x8x3)  established  for  nonresident 
facilities . 

(a)  Commodity  Code.  The  two  alpha  commodity  code  was 
used.  There  are  16  primary  codes  in  the  first  alpha  position  and  up  to  8  in 
the  second  position. 

(b)  QA  Provision.  Three  possible  codes  are  available 
to  describe  the  QA  provision  (MILQ,  MILI  or  standard  inspection). 

(3)  Maintenance  Facilities.  Because  of  the  unique  QA 
conditions  that  exist  in  maintenance  facilities  versus  production  facilities, 
three  separate  groups  were  set  up  for  maintenance  facilities.  Maintenance 
facilities  were  defined  as  those  with  an  Operation  Type  Code  of  "C" 
(overhaul/repair)  or  Commodity  Code  "A3"  for  Aircraft  Programmed  Depot 
Maintenance  commodity.  The  three  maintenance  groups  are  non-resident, 
resident  under  eight  government  QASs  and  resident  eight  or  more  QASs. 

(4)  Any  group  not  assigned  o  one  of  the  previous  groups 
(because  of  missing  or  unexpected  codes)  was  also  assigned  a  "catch-all"  group 
number.  Thus  there  are  a  total  of  628  different  potential  groupings  in  the 
unconstrained  case. 

b.  Constrained  Groups.  Despite  the  fact  that  there  are  several 
thousand  facilities  reported  by  the  five  Phase  II  regions,  most  of  the 
facilities  are  clustered  into  a  few  popular  groupings.  Many  of  the 
unconstrained  groupings  have  few  or  no  facilities  that  meet  the  grouping 
criteria.  Therefore,  these  groups  were  combined  logically  to  ensure  that  each 
peer  group  has  a  reasonable  number  of  "similar"  facilities  to  establish  group 
norms  and  standard  deviations.  Commodity  codes  were  combined  per  logic  shown 
in  Appendix  F. 

c.  When  additional  DCAS  Regions  operate  under  Phase  II,  and  as 
more  data  is  collected  at  the  existing  regions,  it  is  expected  that  some  of 
the  inter  and  intra  commodity  combinations  can  be  re  -  segregated . 
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3.  Creation  of  a  Master  Data  File.  An  integral  part  of  the  QUEST 
model  is  an  external  file  that  contains  historical  statistics  on  QAMIS  data 
elements  broken  out  by  peer  grouping.  This  file  was  developed  by  merging  the 
Phase  II  regions'  records  described  in  paragraph  IIC2.  A  standard  statistical 
package  was  used  to  compute  the  average  and  standard  deviation  for  key  data 
elements  by  group  identification  number.  Rules  used  to  create  the  Master  Data 
File  are: 


a.  The  first  month  of  available  data  from  each  region  was 
discarded  because  of  warnings  that  the  initial  data  was  unreliable.  Note: 
The  master  data  file  was  created  from  data  collected  from  February  1986 
through  April  1987.  Because  Phase  II  was  phased  into  the  regions,  the  number 
of  observations  was  skewed  towards  regions  that  have  been  using  Phase  II 
longer.  The  DCASR-ATL  contribution  to  the  statistics  is  much  greater  then  the 
DCASR-DAL  contribution. 

b.  For  groups  that  had  no  or  few  observations,  a  hypothetical 
record  was  created  by  substituting  a  similar  group. 

(1)  Resident  Facilities.  It  was  observed  that  facility 
size  was  the  dominant  factor  in  most  QAMIS  data  elements.  Therefore  a  size 
group  average  (crossing  commodity  and  provision  boundaries)  was  computed  and 
substituted  for  void  or  sparse  groups. 

(2)  Nonresident.  If  a  group  was  void  or  sparse,  non¬ 
resident  commodity  averages  were  used  for  the  Master  Data  File. 

c.  The  Master  Data  File  was  reviewed  by  the  Study  Advisory 
Group.  Rare  inconsistencies  were  resolved  by  using  size  or  commodity  peculiar 
statistics  in  lieu  of  actual  group  statistics. 

d.  The  Master  Data  File  will  require  periodic  update.  With  more 
facilities  upon  which  the  statistics  are  generated,  and  with  shorter  time  gaps 
between  the  Master  Data  File  and  the  QUEST  measurement  timeframe,  QUEST 
comparisons  become  more  valid. 

E.  Calculation  of  Measures  of  Effectiveness.  Having  grouped  each 
facility  by  assigning  a  group  number  and  having  established  what  is  normal  for 
that  group  via  the  Master  Data  File,  the  next  step  for  the  QUEST  model  is  to 
evaluate  each  facility  for  QA  effectiveness.  Appendix  E  contains  further 
model  documentation. 

1.  Definition  of  OA  Effectiveness.  QA  at  a  facility  is  effective 
when,  compared  with  its  peer  group,  the  facility  demonstrates  that  both: 

a.  its  operations  conform  to  established  C QA  Program  policies 
and  procedures  and; 

b.  through  the  use  of  CQAP,  the  product  conforms  to  the 
contractual  requirements. 
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2.  Program  Indicators.  CQAP  Effectiveness  measures  were  designed 
to  evaluate  QAMIS  data  to  determine  if  the  data  indicate  that  the  QAR  is 
following  established  CQAP  regulations  and  guidance.  The  QUEST  model  reviews 
monthly  data,  seeking  conditions  that  indicate  required  actions  are  not  being 
taken  or  unusual  out-of-tolerence  conditions  exist.  When  the  model  detects 
such  conditions,  an  appropriate  "Red  Flag"  is  generated.  The  CQAP  measure  of 
effectiveness  is  driven  by  the  number  of  "Red  Flags"  generated. 

a.  Red  Flags.  Based  on  brainstorming  sessions  with  Phase  II 
DCASR  personnel  and  input  from  the  Study  Advisory  Group,  17  conditions  are 
tested.  These  conditions  indicate  a  potential  problem  that  may  require 
follow-up  by  the  QAR  or  the  first  line  supervisor.  Presence  of  a  flag  does 
not  always  mean  that  the  QAR  is  not  performing  CQAP  properly.  For  example,  a 
flag  may  be  caused  by  a  data  reporting  error.  Also,  flags  may  occur  because 
of  special  processes  and  conditions  that  exist  that  are  peculiar  to  a  given 
facility.  For  example,  a  facility  that  was  flagged  for  shipping  product 
without  inspection  turned  out  to  be  a  coal  mine  which  required  inspections  to 
be  done  at  a  lab  (which  was  reported  as  another  facility  in  the  QAMIS). 
Therefore,  the  CQAP  measure  is  only  an  indicator  of  potential  effectiveness  or 
ineffectiveness.  Table  1  identifies  the  specific  flags  which  are  further 
described  in  Appendix  G  and  documented  in  Appendix  E-20  through  E-23. 


Table  1 
RED  FLAGS 


A  Lots  or  Units  Rejected  without  Corrective  Action  (QDR) 

B  Intensified  Inspection  without  Corrective  Action 

C  Abnormal  Corrective  Action  Distribution 
D  Corrective  Actions  without  Meeting  with  Contractor 
E  Shipments  without  Product  Verification  Inspection 

F  Shipments  without  Visits  -  Nonresident 

G  No  Procedure  Evaluation  at  MILI  or  MILQ  Facility 
H  Contracts  Received  without  Planning 

I  Lots  Rejected  without  Reinspection 

J  Excessive  Days  to  Close  a  Materiel  Deficiency  Report 

K  Excessive  Net  Quality  Assurance  Letters  of  Instruction 

L  Non  QAR  QDR  Actions 

M  Excessive  System  Indicator 

N  Excessive  Principal  Contracting  Officer  Requests 

0  Excessive  Contract  Administrative  Office  Requests 

P  Excessive  Administrative  Hours 

Q  Work  Not  Performed 

b.  Flag  Counts.  In  addition  to  identification  of  Red  Flags, 
QUEST  counts  the  number  of  flags  generated  per  facility  per  month.  It  is 
assumed  that  all  flags  equally  measure  CQAP  ineffectiveness .  The  model 
computes  a  score,  called  Program  Effectiveness,  based  on  the  premise  that  no 
flags  equals  perfect  Program  Effectiveness  and  that  the  more  flags  generated 
the  lower  the  Program  Effectiveness.  Thus  a  no  flag  facility  is  scored  100%, 
and  points  are  deducted  as  flags  are  counted. 
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(1)  It  was  observed  that  larger  facilities  tended  to 
receive  more  flags  than  smaller  facilities.  Thus  the  penalties  per  flag  were 
greater  for  small  facilities  than  for  large  facilities  to  prevent  bias. 

(a)  Nonresident  Facilities.  The  penalty  per  flag  is 
20%  per  flag.  If  there  are  five  or  more  flags  in  a  given  month,  the  Program 
Score  is  "clipped"  at  0%. 

(b)  Resident  Facilities  (less  than  20  QASs) .  The 
penalty  per  flag  is  15%  except  for  the  first  flag  which  carries  a  10%  penalty. 
This  rationale  for  lower  penalties  per  flag  was  based  on  observations  that 
resident  facilities  were  more  prone  to  flags  because  more  activity  occurs. 
Thus  there  are  more  opportunities  for  problems.  The  10%  penalty  for  the  first 
flag  was  justified  by  the  high  percentage  of  facilities  that  get  a  single 
flag.  The  SAG  felt  that  a  single  flag  should  result  in  a  reasonably  good 
Program  Effectiveness  score.  If  7  or  more  flags  occurred,  the  score  was 
clipped  to  0%. 


(c)  Resident  Facilities  (more  than  20  QASs).  It  was 
observed  that  very  large  facilities  (DCAS  Plant  Representative  Offices 
(DCASPRO))  tended  to  receive  more  flags  then  other  resident  facilities.  To 
normalize  the  scores,  the  penalties  were  adjusted  to  be  15%  per  flag  except 
for  the  first  two  flags  which  carry  only  a  10%  penalty.  The  score  was  clipped 
to  0%  for  8  or  more  flags . 

(2)  Table  2  summarizes  the  Penalty  function  used  by  QUEST 
to  compute  Program  Effectiveness. 

Table  2 

RED  FLAG  PENALTY  FUNCTION 


Program  Effectiveness  % 


Number 

Resident 

Resident 

of  Flags 

Nonresident 

less  than  20 

over  20 

0 

100 

100 

100 

1 

80 

90 

90 

2 

60 

75 

80 

3 

40 

60 

65 

4 

20 

45 

50 

5 

0 

30 

35 

6 

0 

15 

20 

7 

0 

0 

5 

8  + 

0 

0 

0 
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3.  Product  Indicators.  Unlike  the  program  indicators  discussed  in 
the  previous  paragraph,  product  indicators  based  on  counts  or  count  ratios  are 
continuous,  monotonic,  and  negative  in  value.  In  other  words,  as  the  count  or 
ratio  increases  for  the  identified  product  indicators,  the  implied  quality  of 
the  end  product  is  believed  to  be  lower.  Unlike  red  flags,  then,  which  are 
either  present  or  not  present,  product  indicators  can  be  quantified  on  a 
continuous  scale  and  readily  combined  using  multi-attribute  techniques.  The 
process  of  evaluation  of  product  quality  is  explained  in  the  following: 

a.  Identification  of  Product  Indicators.  Seven  indicators  were 
identified  by  experts  during  brainstorming  sessions  and  SAG  meetings.  Experts 
indicated  that  not  only  is  the  magnitude  of  the  indicator  important  in 
assessing  product  quality,  but  the  rate  of  change  or  Lrend  is  equally 
important  in  measuring  effectiveness.  The  indicators  selected  for  use  in 
measuring  product  quality  effectiveness  are  shown  in  Table  3  and  further 
discussed  in  Appendix  H. 


Table  3 

PRODUCT  INDICATORS 


Estimated  Process  Average  EPA 

Lot  Rejection  Ratio  LRR 

Material  Review  Board  MRB 

Waivers  and  Deviations  W/D 

Engineering  Change  Proposals  ECP 

Corrective  Actions  CA 

Materiel  Deficiency  Reports  MDR 


b.  Normalization  of  Indicator  Values.  QAMIS  data  was 
normalized  on  a  scale  of  -3.0  to  +3.0  representing  (roughly)  the  number  of 
standard  deviations  from  average  as  outlined  in  Appendix  H. 

c.  Calculation  of  Individual  Indicator  Scores 

(1)  Technique  for  Order  Preference  by  Similarity  to  Ideal 
Solution  (TOPSIS)  [2],  The  TOPSIS  technique  developed  by  Dr.  Ching-Lai  Hwang 
of  Kansas  State  University  is  a  multiple  attribute  decision  making  technique. 
TOPSIS  enables  the  analyst  to  deal  with  several  variables  that  contribute  to 
an  overall  objective  by  combining  the  multiple  variables  into  a  single  factor 
in  a  logical  and  common-sense  manner.  Basically,  TOPSIS  works  by: 

(a)  Defining  a  set  of  positive  ideal  and  negative  ideal 
conditions  for  the  attributes  involved.  In  other  words,  the  best  and  worst 
possible  cases  are  established. 
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(b)  Measuring  any  set  of  attributes  in  terms  of 
distances  from  both  the  positive  ideal  and  negative  ideal  points.  A  ratio  of 
distances  is  computed  to  produce  a  relative  score.  This  ratio  is  the  distance 
from  a  given  point  to  the  negative  ideal  point  divided  by  the  total  distance 
from  the  point  to  both  extremes . 

(c)  Therefore,  if  a  point  is  far  from  the  negative 
ideal  and  close  to  the  ideal  point,  the  ratio  of  distances  is  large  and  the 
score  is  high.  On  the  other  hand,  if  a  point  is  near  the  negative  ideal,  the 
ratio  of  distances  is  small  and  the  relative  score  is  low. 

(2)  Definition  of  "Ideal"  and  "Negative  Ideal"  states.  When 
experts  were  pressed  to  define  an  ideal  set  of  conditions  for  the  seven 

indicators,  a  common  response  was  "It  depends  . "  The  primary  factor  in 

determining  whether  or  not  a  certain  indicator  value  was  "good”  or  "bad" 
seemed  to  be  the  nature  of  the  contractor.  The  interpretation  of  QAMIS  data 
from  a  contractor  facility  that  has  a  history  of  quality  excellence  versus  a 
contractor  that  has  a  poor  track  record  is  different.  For  example,  a  sudden 
upturn  in  corrective  action  activity  for  a  historically  good  contractor  could 
indicate  that  there  are  problems  developing  within  the  contractor's  program  or 
that  the  QAR  is  unfairly  harassing  the  contractor.  Both  of  these 
possibilities  are  negative  and  should  result  in  an  effectiveness  degradation. 
On  the  other  hand,  the  same  data  concerning  a  "problem"  facility  would 
indicate  that  the  QAR  is  finding  and  correcting  problems,  probably  resulting 
in  improved  product  quality. 

(a)  The  criteria  to  identify  whether  or  not  a 
contractor  is  a  problem  contractor  are  presence  on  the  Alert  list  (see 
paragraph  IIA4) ,  high  MDR  counts  and  serious  corrective  actions.  Specifics 
are  provided  in  Appendix  H. 

(b)  If  the  contractor  is  "normal."  The  definition  for 
ideal  and  negative  ideal  conditions  are  the  same  for  all  seven  indicators. 
Ideally  all  indicators  are  three  or  more  standard  deviations  below  average  and 
decreasing  at  rates  of  3  or  more  standard  deviations  (  Z  -  -3  and  -3  for 
indicator  value  and  rate) .  Negative  ideal  is  the  opposite  state  (Z  -  +3  and 
+3). 

(c)  If  the  contractor  is  a  "problem"  contractor,  it  is 
the  QAR's  job  to  try  to  improve  the  contractor's  quality  system.  Therefore 
the  rate  of  change  of  the  indicators  tends  to  be  dominant  or  more  important 
than  the  indicator  values  themselves.  In  other  words,  it  is  already 
recognized  that  there  are  serious  quality  deficiencies  in  the  facility,  so  the 
emphasis  becomes  to  take  action  to  get  better.  For  EPA,  LRR,  MRB,  W/D  and  ECP 
indicators,  ideal  is  a  rate  decrease  of  three  or  more  standard  deviations  and 
negative  ideal  is  a  3  standard  deviation  or  more  rate  increase.  For 
Corrective  Action  (CA) ,  the  rationale  explained  in  paragraph  IIE3c(2)  results 
in  a  definition  of  ideal  CA  to  be  +3  standard  deviations  above  normal.  For 
problem  facilities,  the  QAR  should  be  issuing  many  corrective  actions,  thus 
preventing  product  quality  problems  Negative  ideal  CA  definition  is  when  CA 
counts  are  three  standard  deviations  or  more  below  average.  With  a  known 
problem  contractor,  absence  of  sufficient  QDR  activity  may  indicate  that  the 
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QAR  is  failing  to  assert  control.  Material  Deficiency  Reports  (MDR)  are  the 
most  directly  linked  indicator  to  end  product  quality.  As  such,  MDR  counts 
and  not  rates  are  felt  to  more  reflect  the  product  quality.  Even  if  the  other 
indicators  are  decreasing,  a  continued  high  valid  MDR  count  is  viewed  as  a 
failure  of  the  QAR  and  the  contractor  to  ensure  product  compliance.  Thus  Z 
values  of  -3  standard  deviations  from  average  are  ideal  for  MDR's  for  problem 
contractor.  Z  values  of  +3  are  negative  ideal  for  MDR  counts  at  problem 
facilities.  Appendix  H  contains  additional  information  on  ideal  and  negative 
ideal  states. 


d.  Calculation  of  overall  Product  Score.  TOPSIS  enables  the 
analyst  to  mathematically  combine  the  individual  attributes,  weighted  by 
relative  importance. 

(1)  Weight  factors  were  developed  by  the  SAG  using  SPAN 
[1}  methodology.  Weights  for  resident  facilities  chosen  were  EPA  -  .77; 
LRR  -  .77;  MRB  «  .91;  W/D  -  .72;  ECP  -  .32;  CA  -  1 . 00 ;  and  MDR  -  .98. 
For  nonresident  facilities,  the  weights  are  EPA  -  .66;  LRR  -  .98;  MRB  -  .83; 
W/D  -  .64;  ECP  -  .32;  CA  —  1.00;  and  MDR  -  .98. 

(2)  Overall  product  score  is  computed  using  the  ratio 
of  Euclidean  weighted  distances  in  14  dimensional  space  (7  dimensions  for 
problem  facilities)  as  shown  in  Appendix  H. 

4.  Total  Scores.  A  final  numerical  score  is  computed  by  taking  a 
weighted  average  of  the  program  and  product  scores.  The  program  score  is 
weighted  60%  and  the  product  score  was  weighted  40%.  Weights  were  provided  by 
the  SAG  using  SPAN  methodology  [1).  This  total  score  is  also  called  the  QUEST 
score . 


F.  Report  Generation.  The  final  module  of  the  QUEST  model  produces  a 
hard  copy  printout  detailing  the  monthly  effectiveness  scores  produced  by 
QUEST.  After  QUEST  has  computed  the  measures  of  effectiveness  for  all 
facilities  in  the  target  region  during  the  months  specified  by  the  Program 
Control,  an  output  file  is  created.  This  file  is  sorted  by  year,  month, 
organization  codr,  facility  type,  and  QUEST  score.  This  file  is  the  data 
source  for  the  report  generator.  The  purpose  of  the  report  generator  is  to 
produce  an  informative,  one  page  summary  for  each  section  and  to  compute 
average  measures  at  various  levels  of  the  organization.  Table  4  is  a  sample 
of  the  report.  Appendix  I  contains  further  documentation. 

1.  Report  Characteristics.  The  following  information  and 
corresponding  values  shown  in  Table  4  are  reported. 

a.  Date.  The  month  and  year  of  the  QAMIS  data  used  to  measure 
QUEST  scores  appear  in  the  upper  left  corner;  i.e.  7-87  representing  July  1987 
measurement . 
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Sample  Output 
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b.  QA  Organization  Code.  The  three  alpha  QA-Org  Code  appears 
in  the  top  line  and  in  the  subtotal  line  following  the  dashed  line.  This  data 
element  identifies  the  division  (first  alpha),  branch  (second  alpha)  and 
section  (third  alpha).  For  example  All  represents  Division  A  (DCASMA 
Atlanta),  Branch  1  (Atlanta  Operations),  Section  1  (Marietta)  within  DCASR 
Atlanta . 

c.  FSCM.  Identification  of  facilities  is  by  FSCM.  FSCM's 
above  the  blank  line  are  resident  facilities.  Inactive  facilities  are  not 
reported.  If  there  were  no  hours  expended  against  an  active  facility  in  a 
particular  month,  no  line  will  appear  for  that  month.  If  an  asterisk  (*) 
appears  before  the  FSCM,  the  facility  was  identified  as  a  "problem"  facili  ty 
by  QUEST  as  described  in  Appendix  H.  QUEST  limits  the  number  of  active  FSCM's 
per  section  to  500  per  month. 

d.  GRP.  The  peer  group  number  is  shown  for  each  FSCM.  For 
example,  FSCM  90454  was  assigned  to  group  152,  a  resident,  nuclear,  MIL-Q 
facility  with  3  to  7  government  QAS's  in  residence  per  Appendix  F. 

e.  Red  Flags.  Individual  red  flags  identified  are  shown  by 
letter.  For  example,  in  July  1987,  FSCM  90454  received  three  flags  (C,  P  and 
Q)  resulting  in  the  following  Program  Score. 

f.  Program  Score.  The  number  of  flags  generated  produces  a 
Program  Score  in  accordance  with  Table  2.  FSCM  90454,  a  resident  facility 
with  less  than  20  QAS's,  is  reported  as  60%  effective  on  its  program  score  (a 
very  poor  rating) . 

g.  Individual  Attribute  Scores.  Each  of  the  seven  product 
related  indicators  are  scored  using  the  TOPSIS  methodology  and  reported.  For 
example,  FSCM  90454  scored  very  high  concerning  EPA  and  MRB  but  scored  very 
poorly  concerning  waivers  and  deviations.  Generally  speaking,  individual 
product  scores  over  70%  are  usually  above  average. 

h.  Product  Score.  A  TOPSIS  generated  "weighted  average" 
combining  the  individual  product  indicators  scores  is  reported.  The  product 
score  of  64%  for  FSCM  90454  is  slightly  below  average. 

i.  Total  Score.  Combining  the  program  score  and  the  product 
score  by  a  60/40  ratio  respectively  produces  the  total  score.  Within  facility 
type,  the  sequence  of  FSCM's  reported  is  in  ascending  order  of  Total  Score. 
The  QUEST  rating  of  61.6%  for  FSCM  90454  is  a  relatively  low  rating,  caused 
primarily  by  the  low  program  score. 

j.  Prior  Month,  The  previous  QUEST  rating  produced  in  the 
prior  month  of  record  is  shown.  For  example,  FSCM  90454  scored  a  74.8%  Total 
Score  in  June  1987.  The  reason  for  the  sudden  drop  in  ratings  from  June  to 
July  was  the  appearance  of  the  P  flag  (Admin  Hours)  and  a  very  large  number  of 
waivers  and  deviations  received  in  July. 
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k.  Peer  Rating.  In  addition  to  the  ability  to  compare  FSCM’s 
within  sections,  the  users  desired  the  ability  to  evaluate  a  FSCM  against  its 
peers  DLA-wide.  To  simplify  the  comparison,  a  letter  grade  of  A,  B,  C,  D  or  F 
is  assigned  based  on  a  comparison  of  the  Total  Score  with  a  Peer  Group  average 
Total  Score.  Since  FSCM  09454  obtained  a  61.6%  Total  Score  and  because  Group 
152  facilities  DLA-wide  average  84%  with  a  standard  deviation  of  7%,  this 
facility  is  3.2  standard  deviations  below  average.  This  parameter  equates  to 
the  letter  value  "F."  Each  peer  group  has  its  own  set  of  average  and  standard 
deviations.  Some  peer  groups  tend  to  be  more  "ineffective"  than  others  on  the 
average.  Thus  a  score  of  90%  for  FSCM  27253  (nonresident,  general,  lumber  and 
wood  products,  standard  inspection)  rates  only  a  "C”  because  historically, 
this  group  has  a  high  average  (86.6%)  whereas  a  90%  for  most  groups  would 
result  in  a  "B"  or  "A"  rating. 

2.  Sectional  Subtotals.  Averages  by  Facility  type  were  computed 
for  each  section  for  all  percentage  scores  discussed  in  the  preceding 
paragraph. 


a.  Resident.  The  resident  subtotal  is  the  weighted  average  of 
resident  facilities  reported  during  the  month  of  record  at  the  section.  The 
weight  assigned  is  roughly  proportional  to  the  size  of  the  facility  measured 
by  the  number  of  government  QASs .  The  relative  weights  assigned  are  0-2 
QASs  -1.0;  3-7  QASs  -  4.0;  8-20  QASs  -  9.0  and  over  20  QASs  -  25.0. 

b.  Nonresident.  A  simple  average  is  computed  for  all  scores 
for  all  nonresident  facilities  within  the  section. 

c.  Combined.  The  weighted  overall  average  for  all  facilities 
is  computed.  Individual  nonresident  facilities  are  weighted  at  0.2  (5 
nonresident  facilities  equals  one  small  resident  facility). 

3.  Higher  Level  Averages.  Scores  are  also  averaged  at  the  branch, 
division  and  regional  levels  by  accumulating  individual  facility  scores  at 
each  level  using  the  same  weighting  factors  used  to  compute  sectional 
averages.  In  the  report,  the  branch  averages  appear  at  the  end  of  the  last 
section  of  the  branch.  Likewise,  the  division  averages  appear  at  the  end  of 
the  last  branch.  The  regional  averages  are  found  on  the  last  page  of  the 
report  for  the  month  of  record. 

III.  VALIDATION .  The  SAG  decided  that  it  was  necessary  to  verify  that  the 
QUEST  model  produces  reasonably  reliable  indicators  of  effectiveness. 
Unfortunately,  there  are  no  measures  of  effectiveness  that  are  currently  being 
used  to  compare  against  the  QUEST  results.  Therefore,  validation  was 
attempted  by  comparing  the  model  with  expert  opinion.  Questionnaires  were 
sent  to  QA  Program  Experts  at  all  echelons  from  Headquarters  DLA  down  to 
section  chiefs  in  DCASRs  ATL,  CHI  and  STL.  Each  echelon  was  asked  to 
evaluate  the  effectiveness  and  effectiveness  trend  of  each  subordinate 
activity  on  scales  of  0-100.  Questionnaire  results  were  converted  to  ordinal 
rankings.  These  rankings  were  compared  with  the  corresponding  QUEST  rankings 
using  a  statistical  technique  called  the  Spearman  Rank  Correlation  Test  [3], 
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A.  Initial  comparisons  were  mostly  favorable  but  several  instances  of 
high  disagreement  between  model  and  expert  were  observed.  Through  a  process 
of  follow-up  talks  with  experts,  questionnaire  clarification,  model  changes 
and  revalidation,  the  revised  results  of  the  tests  are  shown  in  Table  5.  The 
major  factors  in  disagreement  found  were  misinterpretation  of  the 
questionnaire  and  the  use  of  data  external  to  QAMIS  by  the  expert  to  evaluate 
effectiveness . 


Table  5 

AVERAGE  CORRELATION  OF  QUEST  TO  EXPERTS 


Spearman  Rank  Correlation-R 


Avg 

Prob  of 

Avg 

Prob  of 

Sub  Org 

Avg  # 

Relative 

Coincidental 

Trend 

Coincidental 

Expert 

Evaluated 

of  Subs 

Eff  R 

Correlation 

R 

Correlation 

DLA-Q 

DCASR 

5 

.500 

16% 

.600 

12% 

DCASR's 

Divisions 

9 

.366 

13% 

.265 

22% 

Divisions 

Branches 

5 

.485 

16% 

.443 

19% 

Branches 

Sections 

5 

.267 

30% 

.300 

27% 

Section 

Facilities 

15 

.441 

5% 

.265 

16% 

Average 

8 

.412 

13% 

.375 

15% 

B.  If  the  model  produced  the  same  ranking  of  subordinates  as  the 
expert,  the  R  values  shown  would  be  +1.0.  If  the  model  exactly  inverted  the 
expert's  rankings,  R  would  be  -1.0.  A  correlation  of  zero  implies  no 
correlation,  neither  favorable  or  contrary.  Whether  or  not  a  given  value  of  R 
is  significant  depends  on  the  number  of  items  ranked.  For  example,  a  R  value 
achieved  by  ranking  5  DCASRs  yielding  R  -  .5  could  occur  by  coincidence  16% 
of  the  time.  The  same  .5  R  value  achieved  by  ranking  9  DCASBs  could  only 
occur  by  coincidence  about  7%  of  the  time. 

C.  The  correlation  with  expert  opinion  shown  in  Table  5,  although  not 
extremely  high,  is  consistently  positive  and  consistent  through  the  echelons 
of  the  organization.  In  several  cases,  the  experts  indicated  that  they  had 
more  confidence  in  the  model  than  their  own  assessments. 

IV.  BENEFITS .  The  SAG  was  asked  to  provide  a  list  of  potential  benefits  of 
implementing  QUEST  at  various  levels  of  the  organization.  Results  are  shown 
in  Table  6.  In  summary,  QUEST  provides  measures  of  QA  effectiveness  that  are 
currently  not  available.  These  measures  are  systematic  and  informative  and 
they  provide  QA  personnel  with  insight  regarding  potential  problems  and  trends 
for  early  resolution.  QUEST  could  reduce  the  burden  of  first  line  supervisors 
and  above  concerning  review  of  QAMIS  data  and  the  burden  of  Quality  Data 
Evaluation  for  some  QARs . 


Table  6 


POTENTIAL  BENEFITS  OF  QUEST 


For  Section  Chiefs 

1.  Provides  a  quantifiable  measure  of  effectiveness  of  the  QARs  application 
of  CQAP . 

2.  Provides  a  quantifiable  measure  of  contractor  effectiveness. 

3.  Concise  statistical  summary  of  QAMIS  data.  Should  result  in  less  time 

|  spent  reviewing  QAMIS  printouts. 

4.  Helps  identify  problem  areas  for  staff  assistance  and  training  needs. 

5.  Resource  allocation  tool.  Could  be  used  to  justify  requests  for 
additional  resources. 

6.  Tool  for  new  supervisors  to  get  oriented. 

7.  Facilitates  comparisons  within  section  and  allows  visibility  on  how 
section  compares  with  peers  DLA-vidc. 

8.  Trend  becomes  more  visible. 

I 

For  Branch  Chiefs 

1.  All  of  the  above. 

|  2.  Provides  a  means  to  validate  FLS  reviews. 

3.  Provides  information  on  individual  FSCMs  in  a  concise  form. 

4.  Provides  an  objective  means  to  compare  section  performance. 

For  Division  Chiefs 

1.  All  of  the  above  * 

2.  Supports  decision  on  where  to  provide  staff  support  and  audits. 

3.  Vehicle  of  communication  with  other  functional  areas.  QAMIS  data  is  in  a 
form  that  non-QA  personnel  can  relate  to. 

4.  Assists  and  validates  Pre-Award  Surveys.  Could  result  in  more  desk 
audits . 
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For  Regional  Staff 

1.  All  of  the  above.  * 

2.  Determine  where  policy  guidance  and  clarification  needed. 

3.  Identifies  areas  needed  for  new  training  programs. 

4.  Identifies  commodity  related  problems  for  corrective  action. 

For  DLA-O  Staff 

1.  All  of  the  above.  * 

2.  Identifies  areas  for  Management  Reviews. 


*  Use  of  QUEST  by  high  level  managers  for  some  of  the  beneficial 

applications  of  lower  level  managers  could  be  a  negative  benefit  of  QUEST. 
Section  and  Branch  Chiefs  (as  well  as  brainstorming  team  members  during  summer 
1986  meetings)  have  repeatedly  warned  that  they  are  apprehensive  that  QUEST 
will  be  used  to  micromanage  CQAP .  QUEST  could  be  a  burden  on  the  lower  level 
managers  if  they  have  to  respond  to  frequent  inquiries  about  specific 
effectiveness  problems  on  a  monthly  basis. 
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V. 


A.  The  QUEST  model  measures  the  effectiveness  of  QA  as  defined  in  this 
study.  The  model  produces  both  detailed  and  summary  level  measures  from 
indicators  available  in  the  QAMIS. 

B.  The  QUEST  model  is  valid.  It  was  developed  based  extensively  on 
input  from  experts  and  has  been  successfully  tested. 

C.  The  QUEST  model  has  significant  potential  benefits.  It  provides  the 
manager  with  new  and  useful  information.  QUEST  could  improve  the  productivity 
of  QA  Supervisors  and  could  ease  the  Quality  Data  Evaluation  burden  on  some 
QARs.  However,  if  QUEST  is  used  by  managers  at  high  levels  to  micro-manage 
or  to  burden  lower  level  managers  with  inquiries,  it  may  do  more  harm  than 
good. 


D.  The  QUEST  model  will  require  periodic  updates.  Significant  policy 
changes,  changes  to  QAMIS,  new  regions  using  Phase  II  and  general  trends  could 
cause  certain  portions  of  QUEST  to  be  out-of-date. 

VI.  RECOMMENDATIONS  AND  IMPLEMENTATION 


A.  It  is  recommended  that  DLA-Q  approve  the  SAG  plan  to  implement  QUEST 
on  a  trial  basis  at  one  or  more  Phase  II  regions  for  a  four  to  six  month  test. 
This  test  will  require  assistance  from  the  DLA  Systems  Automation  Center, 
DSAC .  If  the  QUEST  model  is  determined  to  be  cost  effective,  by  the  SAG,  then 
further  implementation  should  follow: 

1.  All  Phase  II  DCASRs  should  implement  QUEST  immediately. 

2.  DCASRs  BOS,  LA,  NY  and  PHI  should  implement  QUEST  six  months 
following  Phase  II  conversion. 

3.  A  Systems  Change  Request  should  be  initiated  to  DSAC  to 
incorporate  (reprogram)  QUEST  into  QAMIS.  Outputs  from  QUEST  should  become  a 
part  of  the  QAMIS  performance  history  file  in  the  future. 

4.  The  QAMIS  element  called  Performance  Indicator  (PI)  should  be 
renamed  Productivity  Indicator  to  avoid  confusion  with  QUEST. 


5. 

to  abuse  the 
B.  It 
1. 

can  be  made . 


If  QUEST  is  implemented,  managers  must  avoid  a  natural  tendency 
tool . 

is  recommended  that  DLA-LO: 

Continue  to  support  the  QUEST  model  until  a  transition  to  DSAC 


2.  Assist  DLA-Q  in  testing  QUEST  in  future  installations. 
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C.  The  new  QAMIS  offers  opportunities  for  further  analysis  of  CQAP.  It 
is  recommended  that  further  research  be  sponsored  in  the  following  areas:  I 

1.  Develop  a  model,  which  uses  the  QUEST  process,  such  as 
stratification,  problem  facility  identification,  etc.,  to  analyze  efficiency 
measures . 


2.  Establish  relationships  between  resources,  efficiency  and 
effectiveness.  Using  historical  data,  determine  the  marginal  effectiveness 
associated  with  varying  workloads  and  resources. 

3.  Develop  CQAP  guidelines  which  optimize  effectiveness.  For 
example,  for  a  given  grouping  of  facilities,  the  percentage  of  time  allocated 
to  CQAP  elements  that  achieves  the  highest  QUEST  scores  could  be  used  by  QARs 
to  decide  the  appropriate  balance  between  Planning,  PE  and  PVI . 
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organization 

fscm 

type  facility 
primary  commodity 
primary  provision 
reporting  date 
planning  hours 
lots  inspected 

LOTS  REJECTED 
UNITS  INSPECTED 

units  defective 
inspection  hours 
pe  locations  evaluated 

PE  HOURS 

procedures  reviewed 
procedure  review  hours 

QDR  TYPE  A 
OOR  TYPE  B 
QDR  TYPE  c 
QOR  TYPE  D 
QDR  TYPE  E 
TOTAL  QDR  HOURS 
TRAVEL  HOURS 

training  hours 
SUPPLEMENTAL  hours 
FMS  DOLLARS  SHIPPED 
FMS  HOURS 


ADMINISTRATIVE  hours 
number  of  shipments 

NUMBER  OF  WAIVER  DEVIAT 
WAIVER  DEVIATION  HOURS 
INTENSIFIED  INSPECT  HOU 
REINSPECTION  HOURS 
nonresident  VISITS 


DC AS R-CD 

QA-ORG-CD 

FSCM 

TY-FAC-CD 
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MRB  HOURS 
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ECP  REVIEW  HOURS 
HDR  HOURS 
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contractor  meeting 
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MTA  ELEMENT  e-IELD  NAMES 
FACILITY  PROFILE  FILE 


dasc-d  data  names 

COLUMBUS  INPUT  SCREEN 

OCASR 

ORGANIZATION 

DCASR-CO 

fsch 

QA-ORG-CD 

TYPE  FACILITY 

FSCH 

PRIMARY  COMMODITY 

TY-FAC-CD 

PRIMARY  PROVISION 

qa-cmdty-cd 

CONTRACTOR  QA  PERS 
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"Valid"  MDR's  are  defined  as  all  MDR's  that  are  completed  (Action  Code  "F"), 
not  for  information  only  (Priority  Code  -  5)  and  deficiency  is  attributable  to 
CQAP  (Defect  codes  -  A  through  M)  .  Reopened  and  reclosed  MDR's  are  not 
counted  as  new  valid  MDR's,  but  some  of  the  data  from  the  reopened  MDR  is 
moved  to  the  original  MDR.  For  example,  if  upon  reinvestigation,  the  defect 
code  changes  from  "U*  to  "A,”  the  original  MDR  record  will  become  "valid." 
The  days  to  close  a  reopened  MDR  are  added  to  the  days  to  close  the  original 
MDR . 

Because  there  is  a  time  gap  associated  with  CQA  actions  and  MDR  receipt, 
consideration  is  given  to  the  approximate  "age"  of  the  MDR.  "Age"  represents 
an  estimate  of  time  interval  from  ship  date  to  MDR  receipt  date.  An  MDR  for  a 
recent  contract  carries  more  weight  than  a  MDR  for  a  five  year  old  contract 
because  the  CQA  conditions  that  resulted  in  the  recent  contract's  MDR  probably 
still  exist  whereas  "old”  MDR's  probably  were  caused  by  conditions  that  have 
since  been  corrected.  Age  is  computed  by  the  MDR  processor  by  subtracting  the 
contract  year  (from  the  contract  number)  from  the  year  the  MDR  was  received. 
For  MDR's  with  missing  or  invalid  contract  numbers,  the  age  is  assumed  to  be 
0.  Weighting  factors  were  provided  by  the  Study  Advisory  Group  (SAG)  using 
the  Social  Participatory  Allocative  Network  (SPAN)  Technique  [1]  to  elicit 
expert  opinions.  A  list  of  Study  Advisory  Group  Members  is  provided  in 
Appendix  C.  Results  are  given  in  Table  C-l. 


Table  C-l 

MDR  WEIGHTING  FACTORS 


Age  of  MDR  (Yr)  Weight 

0-1  1.00 

2  .82 

3  .47 

4  .18 

5  +  .00 


The  MDR  processor  creates  an  MDR  record  sorted  by  FSCM  by  year  received  by 
month  received  in  ascending  order.  The  module  sums  the  weighted  number  of 
valid  MDR's  received  during  the  month  by  the  FSCM  and  computes  the  average 
days  to  close  the  MDR. 

For  example,  suppose  a  facility  received  five  MDR's  in  a  particular  month  as 
shown  in  Table  C-2.  The  MDR  processor  would: 

1.  exclude  the  first  and  third  MDR  because  of  "invalid"  action  code  and 
defect  code  respectively. 


C-2 


2.  replace  the  contract  number  In  the  second  MDR  with  the  contract 
number  shown  in  the  reopened  MDR  and  add  the  days  to  close  the  reopened  MDR  to 
the  original  MDR.  Delete  reopened  MDR. 

3.  convert  the  Julian  Date  to  month-year  format. 

4.  compute  the  weighted  number  of  MDR's  and  compute  the  average  day  to 
close.  The  record  created  for  the  FSCM  would  show  for  the  month  and  year  (01, 
87)  a  weighted  MDR  count  of  1.00  +  .47  -  1.47  and  an  average  day  to  close  of 
((30  +  14)  +  (40))  /  2  -  42  days. 

Table  C-2 

EXAMPLE  OF  MDR  FILE 


FSCM 

RCN 

PRI 

DATE 

ACT 

DEF 

DAYS 

CONTRACT 

ABCDE 

SXXXXA870001 

4 

87002 

A 

D 

DLAXXX84 

ABCDE 

SXXXXA870027 

4 

87007 

F 

A 

30 

Unknown 

ABCDE 

SXXXXA870063 

4 

87013 

F 

U 

72 

FXXXXX81 

ABCDE 

SXXXXA875027* 

4 

87110 

F 

A 

14 

DAAXXX86 

ABCDE 

SXXXXA870090 

4 

87027 

F 

B 

40 

NXXXXX84 

*  Reopened  870027 

Because  of  FORTRAN  limitations,  there  are  limits  on  the  MDR  file  that  is 
accessed.  The  number  of  MDR's  must  not  exceed: 

1.  100,000  total  MDR's. 

2.  5,000  "valid"  MDR's. 


3.  100  "valid"  MDR  per  FSCM  per  month. 
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Appendix  E 


The  QUEST  Model  FORTRAN  Source  Code  and  JCL 


Program  Control  File 
Record  Selection  Process 
QUEST  JCL 

QUEST  FORTRAN  Source  Code 


Page 


E-2 


E-3 


E-4 


E-5  -  E-24 
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E-l 
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Program  Control  File 


A  four  record  disk  file  is  accessed  by  QUEST  to  provide  the  following 
parameters.  This  file  is  updated  by  the  computer  operator  prior  to  execution 
of  QUEST.  The  program  code  addressing  this  file  appears  on  pages  E-5  and  E-6. 

First  Record  -  Specifies  Start  Date. 

Position  1-2;  Month  to  begin  Model  measurement  -  2N. 

Position  3-4;  Year  to  begin  Model  measurement  -  2N. 

Second  Record  -  Specifies  End  Date. 

Position  1-2;  Month  to  end  Model  measurement  -  2N. 

Position  3-4;  Year  to  end  Model  measurement  -  2N. 

Third  Record 

Position  1-2;  Minimum  number  of  monthly  data  points  before  the 
start  month  needed  to  have  sufficient  history  to  compute  effectiveness  scores 
-  2N.  Throughout  the  model  development  and  validation  phases  of  this  effort, 
this  parameter  was  set  at  5  months. 

Position  3-4;  Year  before  which  all  historical  data  is  ignored  - 
2N . 


Fourth  Record 

Position  1;  Specifies  whether  or  not  the  Alert  File  will  be  used  by 
the  model.  If  the  Alert  File  is  unavailable  or  not  desired,  position  is  "1". 

Position  2;  Specifies  the  output  products.  A  "0"  or  blank  will 
suppress  all  output  files  except  a  hard  copy  printout  of  the  final  model 
results.  A  "1"  will  cause  intermediate  disk  output  files  to  be  created  at 
various  stages  of  the  model,  enabling  more  detailed  analysis  and  diagnosis,  if 
desired. 


E-2 


Record  Selection  Process 


Records  are  screened  by  QUEST.  Not  all  facilities  or  monthly  records  are 
evaluated  from  the  merged  tape  file  addressed  in  paragraph  IIB. 

Inactivity.  If  no  QAR  hours  were  reported  against  an  FSCM  in  a  given 
month,  the  monthly  record  is  deleted  (see  Page  E-6).  If  there  are  fewer 
months  of  available  history  than  specified  in  the  program  control  file  (see 
Page  E-2),  the  total  history  for  the  FSCM  is  deleted  (see  also  Page  E-6). 
Furthermore,  if  QUEST  is  asked  to  measure  effectiveness  over  a  longer  period 
of  time  than  the  historical  data  will  allow,  a  message  is  generated  and  the 
available  monthly  records  are  moved  to  the  disk  file  but  are  disregarded  in 
subsequent  steps  (see  Page  E-15  and  E-17). 


The  model  checks  certain  key  elements  per  Page 


Date  of  Data.  If  the  data  is  older  than  specified  by  the  Program 
Control  File,  the  record  is  deleted.  If  the  data  is  newer  than  the  end  date 
of  the  model  it  is  likewise  deleted  (see  Page  E-6). 
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Resident  Facilities 


Certain  commodities  were  combined.  The  model  does  not  distinguish  commodity 
differences  for  the  following: 

Aircraft  (A)  and  Missiles/Space  (X) 

Munitions  (B)  and  Weapons  (W) 

Marine  (D)  and  Automotive  (V) 

Electrical  (E) ,  Electronic  Systems  (K)  and  Electronics  (L) 
General  (G)  and  Service  (S) 

Chemical  (H)  and  Petroleum  (P) 

In  the  conversion  table  on  the  next  page,  the  primary  group  number  reflects 
the  initial,  unconstrained  grouping  described  in  Appendix  E-9  through  E-10. 
The  secondary  group  number  reflects  the  above  combinations  per  Appendix  E-ll. 
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Nonresident  Facilities 


A  more  complex  method  was  needed  to  combine  nonresident  facilities  to 
eliminate  voids  in  the  data.  The  following  criteria  are  listed  in  order  of 
precedence : 

1.  MILQ  Facilities.  The  second  alpha  of  the  commodity  code  is 
ignored.  Thus  all  nonresident,  MILQ  facilities  are  combined  based  only  on 
primary  commodity  code. 

2.  MILI  Facilities.  The  second  alpha  of  the  commodity  code  is 
ignored  for  Chemical,  Nuclear  and  Service  commodities. 

3.  Standard  Inspection.  The  second  alpha  for  Nuclear  commodities 
is  ignored.  For  example,  for  nonresident  Nuclear  facilities  there  are  only 
three  actual  groups  (MILQ,  MILI  and  Standard  Inspection). 

4.  Special  Cases.  Combining  commodities  does  not  imply  crossing  QA 
provision  boundaries.  MILI  and  standard  inspection  facilities  are  never 
grouped  together. 

a.  Marine  Repair  commodities  (D2)  were  grouped  with 
nonresident  maintenance  facilities  even  if  Operation  Type  Code  is  not  "C"  . 

b.  Munitions  B1  (Conventional  Explosives)  and  B2  (all 
ammunition)  were  combined. 

c.  Chemical  HI  (military  chemical  agents)  and  H2  (Military 
biological  agents)  were  combined  for  standard  inspection  facilities. 

d.  Facilities  were  combined  for  B3  (fuses),  B4  (Munitions 
metal  parts),  B5  (Munitions,  Miscellaneous)  and  W6  (Miscellaneous  Weapons). 

e.  Facilities  were  combined  for  Clothing  and  Textiles,  C3 
(Footwear  and  leather  goods),  C5  (Embroidered  Insignia)  and  C6 
(Miscellaneous) . 

f.  Electronics  LI  (Electron  Tubes)  and  1.5 
(Miscellaneous)  were  combined. 

g.  Petroleum  PI  (Petroleum  Transportation  and  Storage 
Services),  P4  (Crude  Oil),  P6  (Other  fuels)  and  P7  (Petroleum  "Into  Planes" 
Service)  were  combined. 

h.  Weapons  W2  (Artillery),  W4  (Rocket  Launchers  and  Auxiliary 
Support  Equipment)  and  W5  (Auxiliary  Support  Equipment  and  Weapon  Systems 
Components)  were  combined. 
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i.  Missiles  and  Space  XI  (Guided  Missiles  and  Components),  X2 
(Space  Vehicles  and  Components  including  Satellites)  and  A4  (Aircraft/Airframe 
Structured  Components  and  Subassemblies)  were  combined. 

j .  Missiles  and  Space  X4  (Ground  Support  or  Handling 
Equipment) ,  X5  (Miscellaneous)  and  Aircraft  A3  (Aircraft  Electronic 
Components/Subassemblies)  and  A7  (Miscellaneous)  were  combined. 

k.  Missiles  and  Space  X3  (Instrumentatior/Simulators  and  Control 
Equipment)  and  Aircraft  A6  (Aircraft  Simulators)  were  combined. 

l.  Weapons  W3  (Armored  Vehicles  and  Weapon  Carriers)  and  VI 
(Vehicles  and  Trailers)  were  combined. 

m.  Electrical  E2  (Power  and  Distribution  Equipment)  and  E6 
(Miscellaneous)  were  combined. 

The  above  combinations  are  reflected  in  the  conversion  table  on  the  next  page . 
The  code  that  creates  this  constrained  grouping  is  documented  in  Appendix  E-ll 
through  E-12. 
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Flag  'A.'  "Lots  or  Units  Rejected  without  Corrective  Action."  If,  in  the 
preceding  month  of  record,  there  was  a  lot  rejected  and  there  has  been  no 
corrective  action  taken  of  at  least  type  B  (written)  during  the  prior  month  or 
the  current  month,  the  'A'  flag  is  generated.  If  there  was  no  lot  inspected, 
the  model  performs  a  similar  check  on  units  inspected  and  rejected  during  the 
previous  month.  Thus  for  a  facility  that  did  not  report  lots,  units  rejected 
is  used  in  lieu  of  lots  rejected.  If  there  is  a  reject,  the  QAR  has  the 
current  month  or  the  next  month  to  issue  a  type  B  or  higher  QDR  to  avoid  the 
'A'  flag  in  the  next  month.  Note:  preceding  month  of  record  means  the 
previous  month  or  the  previous  visit  whichever  is  greater.  For  a  non-resident 
facility,  the  preceding  month  of  record  could  be  several  months  in  the  past. 

Usually,  however,  the  preceding  month  of  record  is  the  prior  month. 

Flag  'B.'  "Intensified  Inspection  Without  Corrective  Action."  When 
intensified  inspection  hours  are  reported,  previous  rejections  are  implied. 

Therefore,  corrective  actions  of  a  least  level  B  are  required.  If  the  model 
fails  to  find  at  least  one  type  B  QDR  in  the  month  of  intensified  inspection 
or  the  prior  month  (of  record),  Flag  'B'  occurs.  Caution:  This  flag  could 
be  unfairly  generated  when  the  contractor  takes  a  very  long  time  to  correct 
causes  of  rejection  and  resubmit  items  for  reinspection. 

Flag  'C.'  "Abnormal  Corrective  Action  Distribution."  For  a  given  peer  group, 
a  certain  distribution  of  corrective  actions,  by  type  or  level,  is  expected. 

If  there  are  many  low  level  type  QDR’s  and  few  high  level  types,  this  may 

indicate  that  the  QAR  is  not  properly  escalating  QDR’s.  When,  relative  to 
peers,  a  facility  tends  to  have  lower  level  QDR's  which  are  at  least  one 
standard  deviation  above  average  yet  is  below  average  for  the  next  higher 
level  of  QDR,  Flag  'C'  occurs  as  follows: 

1.  The  average  number  of  Type  A  QDR's  exceeds  plus  one  standard 

deviation  from  normal  and  the  average  number  of  Type  B  QDR's  is  below  normal. 

2.  The  average  number  of  Type  B  QDR's  exceeds  plus  one  standard 

deviation  from  normal  and  the  average  number  of  Type  C  QDR's  is  below  normal. 

3.  The  average  number  of  Type  C  QDR's  is  above  one  standard  deviation 
from  the  group  average  and  the  average  number  of  Type  D  QDR's  is  below  normal. 

Note:  Since  Type  C  and  D  QDR's  are  relatively  rare,  this  flag  will  almost 

always  appear  whenever  Type  C  occurs  and  Type  D  does  not  also  occur  unless  the 
model  is  run  for  several  months  at  a  time.  Note:  Underescalation  flags  tend 

to  recur  because  of  the  averaging  process.  A  facility  may  have  to  alter  its 
QDR  pattern  for  several  months  to  restore  a  proper  balance  and  cause  the  flag 
to  cease. 

Flag  'D.'  "Corrective  Actions  Without  Meeting  with  Contractor."  QAR's  are  ( 

required  to  meet  periodically  with  top  management  of  the  contractor  to  discuss 

CQAP  issues.  If  the  QAR  issues  a  corrective  action  of  at  least  type  B,  the 

QAR  should  meet  with  the  contractor  during  that  month.  If  there  is  at  least 

one  type  B  or  above  QDR  and  there  are  no  meeting  hours  reported  during  the 
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month  of  the  QDR,  Flag  'D'  occurs.  Note:  There  is  no  tolerance  given  for 

timing  of  the  two  events.  It  is  conceivable  that  the  QDR  could  be  written  on 
the  last  day  of  the  month  and  the  meeting  conducted  on  the  first  day  of  the 
next  month.  However  QAR's  are  suppose  to  meet  with  the  plant  manager 
"frequently"  per  DLAM  8200.1  regardless  of  QDR's. 

Flag  'E.'  "Shipments  without  Product  Verification  Inspection."  If  product  is 
shipped,  some  inspection  should  be  made  regardless  of  the  extensive  use  of 
Procedure  Evaluation  (PE)  and  Product  Oriented  Procedure  Evaluation  (POPE) . 
Some  QAR's  mistakenly  believe  that  PE  and  POPE  can  eliminate  the  need 
completely  for  Product  Verification  Inspection  (PVI) .  The  model  will  issue  an 
'E'  Flag  during  any  month  a  shipment  is  made  and  there  have  been  no  inspection 
hours  reported  during  that  month  or  the  prior  month. 

Flag  'F.'  "Shipments  without  Visits-Nonresident . "  If  a  nonresident  facility 
ships  product  during  a  given  month  and  there  is  no  record  of  a  corresponding 
visit  by  the  QAR,  a  Flag  'F'  appears.  A  combination  of  zero  visit  count  and 
non-zero  shipment  count  causes  a  nonresident  facility  to  receive  an  'F'  flag. 
Note:  Many  QAR's  do  not  currently  count  visits  whenever  work  is  done  at  the 

facility  that  is  the  QAR's  duty- station.  To  prevent  this  flag  from 
erroneously  occuring,  visits  must  be  counted  whenever  a  nonresident  performs 
CQA  within  his/her  duty  station.  DLAM  8200.2  requires  these  visits  to  be 
counted  but  many  QAR's  fail  to  do  so. 

Flag  'G.'  "No  Procedure  Evaluation  at  MILI  or  M1LQ  Facility."  If,  for  a 
nonresident  facility,  in  any  two  consecutive  months  or  visits,  whichever  is 
greater,  no  PE  hours  are  recorded  against  a  MILI  or  MILQ  facility.  Flag  'G'  is 
generated.  Resident  MILI  or  MILQ  facilities  must  have  PE  hours  each  month  to 
avoid  the  flag.  This  flag  occurs  in  over  one  in  five  facilities  each  month. 
Note:  This  flag  could  occur  in  active  nonresident  facilities  that  have  a 

small  number  of  special  processes  that  have  90,  120,  180  day  intervals  for  PE. 
Also,  this  flag  could  be  generated  for  active  MILI  and  MILQ  facilities  that 
have  only  standard  inspection  contracts  for  a  period  of  time. 

Flag  'H.'  "Contracts  Received  without  Planning."  If  a  contract  was  received 
during  the  prior  month  and  no  planning  hours  were  reported  for  the  prior  month 
and  the  current  month,  Flag  'H'  occurs. 

Flag  'I.'  "Lots  Rejected  without  Reinspection."  If  at  least  one  lot  was 
rejected  the  prior  month,  and  there  has  been  no  reinspection  hours  reported 
for  the  prior  month  and  the  current  month,  Flag  'I'  is  generated  for  the 
current  month.  If  no  lots  were  inspected,  unit  rejections  are  evaluated  in 
lieu  of  lots.  Note:  This  flag  could  be  caused  by  contractor  delays  In 
submitting  lots  or  units  for  reinspection. 
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Flag  'J.'  "Excessive  Days  to  Close  a  Materiel  Deficiency  Report."  If  the 
average  days  to  close  valid  MDR's  received  during  a  month  exceed  the  group 
average  by  two  standard  deviations,  Flag  'J'  will  be  generated.  Only  MDR's 
deemed  "valid"  per  the  criteria  described  in  Appendix  B,  page  B-2  are 
analyzed.  Thus  open  MDR's  that  are  also  excessive  are  not  flagged.  Depending 
on  the  group,  flags  tend  to  occur  when  the  days  to  close  approach 
approximately  100  days  +  30  days . 

Flag  'K.  '  "Excessive  Net  Quality  Assurance  Letters  of  Instruction  (QALI)." 
High  numbers  of  QALI's  on  hand  are  an  indication  of  lack  of  Principal 
Contracting  Officer  (PCO)  confidence  in  DCAS  CQA.  If  the  QAR  convinces  the 
PCO  to  rescind  a  QALI,  an  indication  of  confidence  is  inferred.  The  model 
computes  a  variable  called  net  QALI  which  subtracts  QALI's  rescinded  from  QALI 
on  hand.  When  this  variable  is  excessive,  compared  to  peers,  Flag  'K'  occurs. 

The  criteria  for  Flag  'K'  is  when  net  QALI  is  more  than  one  standard  deviation 
from  average.  Note:  The  criteria  chosen  to  determine  the  number  of  standard 
deviations  allowed  before  flagging  was  to  use  one  standard  deviation  if  the 
standard  deviation  was  more  than  the  average.  If  the  group  averages  tended  to 
exceed  the  group  standard  deviations,  then  two  standard  deviation  limits  were 
used. 

Flag  'L.'  "Non  QAR  QDR  Actions."  If  the  number  of  corrective  actions  written 
by  someone  other  than  the  QAR  is  above  average  by  one  standard  deviation,  Flag 
'L'  occurs.  Note:  These  events  rarely  occur  or  are  seldom  reported.  Thus 
the  average  and  standard  deviations  for  non  QAR  QDR's  are  very  low.  A  single 
occurrence  is  sufficient  to  generate  this  flag  for  all  groups  but  one  using 
the  existing  Master  Data  File. 

Flag  'M.  '  "Excessive  System  Indicator  (SI)."  High  SI  values  indicate  that 
the  QAR  is  spending  a  large  percentage  of  the  time  on  adverse  actions  such  as 
QDR,  MDR ,  Material  Review  Board  (MRB)  etc.  The  'M'  Flag  occurs  whenever  SI 
exceeds  30%.  Note:  This  flag  should  be  group  dependent  but  reprogramming 
would  be  extensive.  This  action  has  been  deferred  to  planned  future  model 
enhancements . 

Flag  'N.'  "Excessive  Principal  Contracting  Officer  Request."  Frequent  and 
excessive  PCO  requests  are  a  sign  of  lack  of  confidence  by  the  PCO.  When  the 
number  of  requests  exceeds  the  group  average  by  one  standard  deviation,  the 
'N'  flag  is  assigned. 

Flag  '0.'  "Excessive  Contract  Administrative  Office  (CAO)  Requests."  Same  as 
Flag  'N'  concerning  the  number  of  CAO  requests  received  during  the  month  of 
record . 

Flag  ’P.’  "Excessive  Administrative  Hours."  Administrative  hours  is  a  "catch 
all"  category  of  time  reported  to  pick  up  all  time  not  reported  in  other  data 
elements.  If  the  number  of  hours  exceeds  the  group  average  by  two  standard 
deviations,  Flag  P  occurs.  Note:  Administrative  hours  are  a  category  of 
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uncertainty  with  potential  changes  in  definition.  For  example,  Quality  Data 
Evaluation  (QDE)  has  been  reported  as  administrative  time  but  plans  exist  to 
create  a  new  data  element  for  QDE.  Significant  changes  in  any  data  element 
definition  will  cause  problems  with  flag  generation,  requiring  model  update. 

Flag  'Q.'  "Work  not  Performed."  If,  in  any  month,  there  is  any  work  not 
performed  reported,  the  'Q'  flag  will  appear.  The  three  activities  scanned 
for  non- zero  data  are: 

1.  Number  of  product  verification  inspections  not  performed. 

2.  Number  of  procedure  evaluation  locations  not  evaluated. 

3.  Number  of  procedure  evaluation  elements  not  evaluated. 
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DEFINITIONS  OF  PRODUCT  INDICATORS 


Estimated  Process  Average  (EPA) .  EPA  is  defined  as  the  ratio  of  units 
rejected  to  units  inspected.  It  is  reported  in  QAMIS  as  a  percentage. 

Lot  Rejection  Ratio  (LRR)  .  LRR  is  defined  as  the  ratio  of  lots  rejected  to 
lots  inspected.  All  shipments  are  not  inspected  and  reported  in  lots.  Thus 
it  is  possible  to  have  a  positive  EPA  and  a  zero  LRR.  Also  because  of 
Acceptable  Quality  Levels  allowed  in  MIL- STD- 105D ,  it  is  possible  to  reject 
units  yet  accept  the  lot.  Thus  LRR  and  EPA  are  related  but  not  totally 
dependent  indicators . 

Material  Review  Board  (MRB) .  MRB  actions  are  the  number  of  actions  reviewed 
by  the  QAR.  MRB  is  a  mechanism  for  the  contractor  to  obtain  approval  of  minor 
non-conformances  (not  affecting  form,  fit  or  function) . 

Waivers  and  Deviations  (W/D) .  All  waivers  and  deviations  reviewed  by  the  QAR 
are  reported.  A  major  cause  (but  not  the  only  reason)  for  waivers  and 
deviations  is  that  the  contractor  is  unable  to  conform  to  the  existing 
specifications.  The  contractor  can  seek  relief  via  the  waiver/deviation 
process . 

Engineering  Change  Proposal  (ECP) .  ECP  actions  are  a  request  to  change  the 
specification.  ECP  actions  reviewed  by  the  QAR  are  reported.  Some  ECPs  are 
caused  by  the  contractor's  inability  to  meet  specifications. 

Corrective  Actions  (CA) .  The  number  of  Quality  Deficiency  Reports  generated 
by  the  QAR  is  reported  by  type.  There  are  five  types  of  QDRs  ranging  from 
on-the-spot  verbal,  Type  A,  to  termination  of  CQAP,  type  D.  An  artificial 
indicator  was  created  to  produce  a  CA  value  by  taking  the  weighted  value  of 
the  QDR's  by  type.  Weighting  factors  were  produced  by  the  SAG  using  SPAN 
me thodology . 

1.  Type  A  -  Weight  =  3 

2.  Type  B  -  Weight  -  10.  The  SAG  felt  that  a  Type  B  (written) 
corrective  action  was  3  1/3  times  more  serious  than  a  Type  A  QDR.  Thus  three 
Type  A  QDR's  would  almost  generate  the  same  CA  effect  as  a  Type  B. 

3.  Type  C  -  Weight  -  30.  The  SAG  felt  that  a  Type  C  (escalated  to 
higher  levels)  was  3  times  more  serious  than  a  Type  B  QDR.  Thus  3  Type  B 
QDR's  would  generate  the  same  effect  as  a  Type  C. 

4.  Type  D  -  Weight  -  60.  The  SAG  felt  that  two  Type  C  QDRs  should 
carry  the  same  weight  as  a  type  D  QDR. 

5.  Type  E  -  Weight  -  30.  The  SAG  felt  that  a  Type  C  and  a  Type  E 
(subcontractor  corrective  action)  were  roughly  equivalent. 

Materiel  Deficiency  Reports  (MDR) .  The  weighted  MDR  counts  discussed  in 
paragraph  II  B1  represent  the  number  of  valid  MDRs  (or  customer  complaints). 
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NORMALIZATION  OF  PRODUCT  INDICATORS 
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Indicator  values ,  It  was  observed  that  the  distribution  of  product  indicator 
values  tended  to  follow  roughly  the  shape  of  the  exponential  distribution 
rather  than  a  normal,  bell  shaped  distribution.  Counts  and  ratios  were 
normalized  by  computing  a  value  called  a  Z  value  where: 

Z  -  log  indicator  count  or  value 

avg  indicator  count  or  value 

The  average  indicator  count  or  value  is  the  group  average  found  in  the  Master 
Data  File  discussed  in  paragraph.  II  D3 . 

Z  values  less  than  -3.0  or  greater  than  +3.0  were  clipped  to  -3.0  and  +3.0 
respectively . 

For  example,  suppose  a  facility  in  group  number  22  had  an  EPA  of  15%  in  a 
certain  month.  The  average  EPA  for  similar  facilities  is  5.5%.  The  Z-value 
for  that  indicator  value  is  computed  to  be  log  (15/5.5)  -  +1.0033.  (NOTE: 
Log  is  the  natural  logarithm.)  Thus  a  value  of  15%  can  be  expressed  as  being 
approximately  one  standard  deviation  above  normal. 

Indicator  Rates.  The  rate  of  change  was  computed  by  breaking  the  total 
history  of  indicator  values  into  two  equal  groups  from  the  midpoint  of  the 
time  series  of  indicator  values.  If  there  are  more  than  12  months  of  data, 
the  model  ignores  the  earlier  data.  If  there  is  an  even  number  of  monthly 
data,  the  break  is  clean  at  the  n/2th  month.  If  there  is  an  odd  number  of 
monthly  data  points,  the  middle  month  is  excluded  as  belonging  to  neither 
group.  The  average  indicator  value  is  computed  for  the  first  half's  sequence 
and  also  for  the  second  half.  The  rate  is  computed  as  follows: 

Z-3  *  I  Avg  value  for  2nd  half  of  time  series  -3 
I  Avg  value  of  1st  half  of  time  series  J 


Z  values  of  less  than  -3  and  greater  than  +3  are  clipped  to  -3  and  +3 
respectively. 


For  example,  suppose  a  facility  had  seven  observations  for  MRB  counts  up 
a  certain  month.  ,  The  time  series  of  MRB  counts  was  6,  10,  1U ,  9,  10,  5, 
The  Z  value  is 
moderate  rate.  +  ) 


-3  -  -1.2.  Thus  the  indicator  is  decreasing  at 


to 

3. 

a 
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CRITERIA  FOR  IDENTIFICATION  OF  PROBLEM  CONTRACTOR 


Criteria 


1. 

2. 

3. 

4. 

5. 


"At 

Contractor  is  on  the  Alert  list  for  any  cause. 

Contractor  is  more  than  one  standard  deviation  above  average  in 
MDR  count. 

Contractor  has  received  a  Type  C  corrective  action. 

Contractor  has  received  a  Type  E  corrective  action. 

Contractor  once  received  a  Type  D  correction  action. 


*  The  Alert  list  used  by  QUEST  must  not  contain  more  than  2000  FSCM s  .  QUEST 
will  not  execute. 


The  first  criteria  listed  above  will  be  checked  against  the  current  Alert 
list,  regardless  of  the  month  of  record.  For  the  second  criteria,  the  model 
looks  at  the  current  month  of  record  and  prior  month  for  nonresident 
facilities  but  at  the  entire  history  for  resident  facilities.  Thus 
nonresident  facilities  may  jump  back  and  forth  from  "problem"  to  "normal" 
facilities  as  MDR’s  are  received.  Resident  facilities  with  above  average  MDR 
counts  may  take  longer  to  remove  the  label  of  "problem"  and  are  more  stable. 
The  third  and  fourth  criteria  are  also  based  on  the  month  of  record  occurence 
and  will  also  be  "on”  or  "off"  month  to  month.  The  last  criteria  is  either 
always  "on"  or  "off."  The  only  mechanism  to  remove  the  effect  of  a  Type  D 
corrective  action  is  to  wait  until  the  cut-off  parameter  takes  effect  (see 
Appendix  E-2) .  Depending  on  the  criteria  that  activates  the  identification,  a 
facility  may  be  a  "problem"  facility  one  month  and  "normal"  the  following 
month.  This  is  especially  true  for  nonresident  facilities. 

If  a  nonresident  facility  meets  any  of  the  criteria  shown,  it  is  labeled 
"Problem  Facility."  If  a  resident  facility  meets  two  or  more  of  the  criteria, 
it  is  labeled  "Problem  Facility."  Documentation  is  found  in  Appendix  E-12 
through  E-13. 
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Ideal  Facility 

Neg  Ideal  Facility 

Normal 

Problem 

Normal 

Problem 

EPA  Ratio 

-3 

+3 

EPA  Rate 

-3 

-3 

+3 

+3 

1AR  Ratio 

-3 

+3 

LRR  Rate 

-3 

-3 

+3 

+3 

MRB  Count 

-3 

+3 

MRB  Rate 

-3 

-3 

+3 

+3 

W/D  Count 

-3 

+  3 

W/D  Rate 

-3 

-3 

+3 

+3 

ECP  Count 

-3 

+3 

ECP  Rate 

-3 

-3 

+3 

+3 

CA  Count 

-3 

+3 

+3 

-3 

CA  Rate 

-3 

+3 

MDR  Count 

-3 

-3 

+3 

+3 

MDR  Rate 

-3 

+3 

For  a  given  month's  data,  each  product  indicator  Is  scored  by  computing  the 
distances  the  normalized  values  and  rates  are  from  the  ideal  and  negative 
ideal  points.  For  normal  facilities,  QUEST  uses  the  city-block  or  rectilinear 
distance  formula  described  by  Hwang  [2]  for  computing  distances.  Distances 
for  problem  facilities  are  one - dimens ional .  For  example.  Facility  XYZ,  a 
normal  facility,  has  Z  values  of  -2.0  and  +.5  for  a  certain  indicator  in  given 
month.  The  distance  to  the  negative  ideal  point  is  (3- (-2))  plus  (3-. 5)  - 
7.5.  The  distance  to  the  ideal  point  is  (-2-3)  plus  (.5-(-3))  -  U.5.  The 
score  is  the  ratio  of  the  distance  from  negative  ideal  divided  by  the  total 
distance  or f*  7.5  ~l  -  .625  or  62.5%. 


COMPUTATION  OF  PRODUCT  EFFECTIVENESS 


Product  Score 


where  1  -  1 . 14 

or  i  -  1 . 7 

wt^  -  weight  factor  from  SAG 
2 . 

d  £  -  distance  squared  from  negative  ideal 
d  ^  -  distance  squared  from  positive  ideal 
j  -  Knob  for  problem  facilities  -  .79 

(Knob  for  normal  facilities  -  1.00) 

A  "Knob"  of  79%  was  applied  to  the  product  score  of  problem  facilities  to 
reflect  the  expected  lower  product  quality  achievable  under  the  best  of 
conditions.  If  the  contractor  is  "problem,"  the  highest  score  achievable  is 
79%.  The  QAR  can  only  achieve  higher  scores  by  getting  the  contractor 
reclassified  by  QUEST  as  "normal."  This  can  be  done  by  reducing  MDRs , 
eliminating  the  need  for  type  C  and  E  QDRs,  eliminating  the  cause  of  the 
Alert  and/or  waiting  out  prior  Type  D  QDRs.  The  knob  was  provided  by  the  SAG 
using  SPAN  methodology  [1],  Documentation  is  found  in  Appendix  E-18  and  E-19. 
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